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EFFICIENT SPLITTING AND MIXING OF STREAMING-DATA 
FRAMES FOR PROCESSING THROUGH MULTIPLE 
PROCESSING MODULES 

5 

Field of the Invention 

The Present invention relates to electronic data processing, and more specifically 
concerns managing the flow of streaming data through multiple hardware and/or software 
processing modules in a computer. 

1 0 Background of the Invention 

Streaming data is a continuous flow of data that must be ultimately presented to a 
user in a particular sequence in real time. Digital samples representing an audio signal, 
for example, must be converted to a sound wave in the same sequence they were 
transmitted, and at exactly the time spacing they were generated, or some user-specified 
15 altemative. Digital data representing video frames require assembly into the proper 

sequence in the frame for presentation on a display together, and successive frames must 
display at the correct real-time rate. 

Streaming data need not necessarily maintain correct sequence or timing 
throughout an entire communication chain among various transmitters, processors, 

20 memories, and receivers. Indeed, video and audio clips are frequently stored as static 
data in recording media, computer memories, and network buffers. Packet-switched 
systems might also carry parts of the same streaming data over different paths and even 
in different time sequences. Processors such as digital filters can assemble parts of the 
data stream, modify them as a static unit, then release them to further units in the system. 

25 Eventually, however, the stream must be heard or seen in the correct sequence at the 
proper relative times. 

Streaming data almost always involves very large amounts of data. Streaming 
data almost always challenges the capacity of digital buses in computers to access it, 
carry it and switch it. Streaming data almost always taxes the processing power of 



functional units, both software and hardware, to receive it, convert it, and pass it on to 
other units. Those in the art speak of the necessity of "fat pipes" for streaming data. 

An architecture called WDM-CSA (Windows Driver Model Connection 
and Streaming Architecture) introduces the concept of a graph for specifying the 
5 connections among the facilities of a computer where a data stream must pass through a 
number of processing units in an efficient manner. The WDM-CSA protocol also 
simphfies the development of drivers for such data. Basically, WDM-CSA specifies the 
flow of data frames through a graph, and also the control protocols by which adjacent 
modules in the graph communicate with each other to request and accept the data frames. 

10 Commonly assigned patent application "Improving the Flow of Streaming Data 

through Multiple Processing Units," filed on even date herewith (attorney docket 
777.183US1), introduces the concept of data pipes for enhancing the data flow of 
streaming-data frames through a graph of interconnected modules in WDM-CSA and in 
other streaming-data environments. Basically, data pipes avoid redundant storage and 

15 copying of data as a number of modules process the frames, and streamline allocation of 
the frames in which the data is packaged. Another commonly assigned application, 
"Improving the Control of Streaming Data through Multiple processors," filed on even 
date herewith (attomey docket 777.184US1), presents a mechanism for controlling the 
flow of frames through multiple modules by improving the control throughout the entire 

20 graph, rather than by optimizing each individual module separately. These appUcations 
are incorporated by reference. 

In many applications of streaming data, processing could be simpUfied and 
improved by adding capabiHties for spUtting one large frame into multiple subframes and 
for mixing multiple frames together into a single large frame. For example, multimedia 

25 presentations commonly have a single collection of data representing different modalities 
that are to be output together to a user. A frame of television-type data may represent an 
entire field of an NTSC signal, including video data on a number of scan lines, audio 
data, a picture-in-picture frame on parts of some of the scan lines, and data in a video 
blanking interval (VBI). VBI data might include character codes for close-captioning, 

30 digital data representing Web-TV information, and a variable number of other subfields. 
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These frames require complex mixing of data when they are generated, and then must be 
split apart for processing by different hardware or software modules concurrently for 
presentation to a viewer. While the audio data in one subframe is filtered in tone controls 
and compressed/expanded in volume, the video data is resized and color balanced, the 
5 close-captioning is formatted and positioned, and so forth. 

The systems described in the patent appUcations noted above describe primitive 
abilities for spUtting large frames into smaller frames for separate processing. However, 
the growing need for multimedia and similar data streams having multiple types of data 
demands the fixrther ability to combine, merge, or mix multiple frames into a single 
10 frame, and also requires efficient data flow and control of split and mixed frames of 
streaming data. 

Summary of the Invention 

The present invention improves the overall processing of streaming data through a 
15 network or graph of multiple hardware and software processing modules by providing a 
simple and efficient mechanism for splitting a single frame of streaming data into 
multiple frames and for mixing multiple streaming-data frames into a single frame. 

A pipe has multiple interconnected processing modules, one or more of which 
performs an operation that splits, mixes, or otherwise restructures streaming data. A 
20 physically allocated composite frame has subframes that hold different parts of the data 
that are operated upon by different ones of the modules. 

Another aspect of the invention constructs a pipe and assigns an allocator to 
provide the composite frames. A fiirther aspect issues control transactions to a module 
only when all the subframes sourced to that module have become available. Other 
25 aspects concem the construction of tables for specifying the structure of the subframes in 
the composite frame, the structure of the modules in the pipe, and the completion of 
operations by the modules. 
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Brief Description of the Drawing 

Fig. 1 is a block diagram of an illustrative environment for the present invention. 

Fig. 2 details a multi-port memory of Fig. 1. 

Fig. 3 details components of Fig. 1 relevant to the invention. 

5 Fig. 4, comprising Figs. 4A-4C, is a block diagram of an example streaming-data 

pipe handled by the invention, and representations of data flowing therein. 

Fig. 5, comprising Figs. 5A-5E, is a block diagram of another streaming data pipe, 
and representations of alternative composite data frames therein. 

Fig. 6, comprising Figs. 6A and 6B, is a flowchart of a method for setting up and 
10 controlling a graph having composite frames according to the invention. 

Fig. 7, comprising Figs. 7A-7B, shows tables constructed in the method of Fig. 6. 

Fig. 8 shows a fiirther table constructed in the method of Fig. 6. 

Detailed Description 

15 The following detailed description of preferred embodiments refers to the 

accompanying drawings that form a part hereof, and shows by way of illustration specific 
embodiments of the present invention. These embodiments are described in sufficient 
detail to enable those skilled in the art to practice the invention. Structural, logical, and 
procedural modifications within the spirit and scope of the invention will occur to those 

20 in the art. Likewise, the specific forms and sequence in which the description presents 
the hardware, software, and operations of the embodiments does not imply any specific 
interconnections or time order. The following description is therefore not to be taken in a 
limiting sense, and the scope of the inventions is defined only by the appended claims. 



25 Operating Environment 

Fig. 1 is a high-level diagram of an illustrative environment 100 in which the 
invention is implemented as executable instructions, data, and/or hardware on a 
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programmable general-purpose computer such as personal computer (PC) 120. Other 
suitable environments, and variations of the described environment, will occur to those 
skilled in the art, 

A conventional PC 120 typically comprises a number of components coupled 
5 together by one or more system buses 121 for carrying instractions, data, and various 
control signals. These buses may assume a number of forms, such as the conventional 
ISA, PCI, and AGP buses. Some or all of the units coupled to a bus can act as a bus 
master for initiating transfers to other units. 

Processing unit 130 may have one or more microprocessors 131 driven by system 
10 clock 132 and coupled to one or more buses 121 by controllers 133. Internal memory 
system 140 supphes instructions and data to processing unit 130. High-speed RAM 141 
stores any or all of the elements of software 110. ROM 142 commonly stores basic 
input/output system (BIOS) software for starting PC 120 and for controlling low-level 
operations among its components. Bulk storage subsystem 150 stores one or more 
15 elements of software 110. Hard disk drive 151 stores software 1 10 in a nonvolatile form. 
Drives 152 read and write software on removable media such as magnetic diskette 153 
and optical disc 154. Other technologies for bulk storage are also known in the art. 
Adapters 155 couple the storage devices to system buses 121, and sometimes to each 
other directly. Other hardware units and adapters, indicated generally at 160, may 
20 perform specialized fimctions such as data encryption, signal processing, and the like, 
under the control of the processor or another imit on the buses. 

Input/output (I/O) subsystem 170 has a number of specialized adapters 171 for 
connecting PC 120 to external devices for interfacing with a user. A monitor 172 creates 
a visual display of graphic data in any of several known forms. Speakers 173 output 

25 audio data that may arrive at an adapter 171 as digital wave samples, musical-instrument 
digital interface (MIDI) streams, or other formats. Keyboard 174 accepts keystrokes 
from the user. A mouse or other pointing device 175 indicates where a user action is to 
occur. Block 176 represents other input and/or output devices, such as a small camera or 
microphone for converting video and audio input signals into digital data. Other input 

30 and output devices, such as printers and scanners commonly connect to standardized 
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ports 177. These ports include parallel, serial, SCSI, USB, FireWire, and other 
conventional forms. 

Personal computers frequently connect to other computers in networks. For 
example, local area network (LAN) 180 connect PC 120 to other PCs 120' and/or to 

5 remote servers 181 through a network adapter 182 in PC 120, using a standard protocol 
such as Ethernet or token-ring. Although Fig. 1 shows a physical cable 183 for 
interconnecting the LAN, wireless, optical, and other technologies are also available. 
Other networks, such as wide-area network (WAN) 190 can also interconnect PCs 120 
and 120', and even servers 181, to remote computers 19L Fig. 1 illustrates a 

10 communications facility 192 such as a pubUc switched telephone network for a WAN 
190 such as the Internet. PC 120 can employ an internal or external modem 193 coupled 
to serial port 177; however, other known technologies such as ISDN, asynchronous 
transfer mode (ATM), frame-relay, and others are becoming more widespread. In a 
networked or distributed-computing environment, some of the software 110 may be 

15 stored on the other peer PCs 120', or on computers 181 and 191, each of which has its 
own storage devices and media. 

Software elements 110 may be divided into a number of types, whose 
designations overlap to some degree. For example, the previously mentioned BIOS 
sometimes includes high-level routines or programs which might also be classified as part 

20 of an operating system (OS) in other settings. The major purpose of OS 1 1 1 is to provide 
a software environment for executing appUcation programs 1 12. An OS such as 
Windows® from Microsoft Corp. commonly implements high-level appUcation-program 
interfaces (APIs), file systems, communications protocols, input/output data conversions, 
and other fimctions. It also maintains computer resources and oversees the execution of 

25 various programs. Application programs 112 perform more direct fimctions for the user. 
The user normally calls them explicitly, although they can execute implicitly in 
connection with other applications or by association with particular data files or types. 
Modules 1 13 are packages of executable instructions and data which may perform 
fimctions for OSs 1 1 1 or for apphcations 1 12. These might take the form of dynamic 

30 link libraries (.dll). Finally, data files 1 14 includes collections of non-executable data 
such as text documents, databases, and media such as graphics images and sound 
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recordings. Again, the above categories of software 110 are neither exhaustive nor 
mutually exclusive. 

Fig. 2 is a block diagram of one type of conventional memory 200 that is often 
employed in processing high-speed data such as streaming data. Memory module 210 is 

5 a multiport memory 210 having three read/write ports 211-213, each of which can both 
receive data for storage in module 210 and retrieve data already stored there. More or 
fewer ports are possible, and some of the ports can be write-only or read-only. 
Operations at multiple ports can occur simultaneously, although the internal operation of 
module 210 might serialize and/or synchronize them. Memory 200 can form a part of PC 

10 memory system 140, Fig. 1, or might be built into any of the other blocks, such as 

fiinctions 160 or adapters 170. Multiport memories frequently fimction to buffer large 
quantities of data for transfer fi-om one bus to another in a system. Fig. 2 illustrates a 
group of buses 220, which can be included in the group of buses 121 in Fig. 1. Ports 211- 
213 couple to individual buses 221-223, respectively. This connection allows, for 

15 example, a PCI bus to deposit a data frame to memory module 210, and an AGP or 
dedicated video bus to retrieve the frame. 

Handling Composite Frames 

Fig. 3 shows the relevant components 300 that the present invention employs. 

20 For purposes of illustration only, some of the components described are found in the 

Windows-2000 operating system (OS) from Microsoft Corp. Components 310-330 live 
in the kernel layer, although they can reside at other locations in the architecture of a 
particular OS. Although all of the processing in this example occurs in a single 
computer, streaming-data operations according to the invention could be distributed 

25 among multiple machines and/or operating systems. 

Interface component 310 interfaces with other components at the kernel layer, 
with software such as programs 301 and 302 outside the OS, and with hardware devices 
such as devices 303-304 and hardware adapter 305. Application program 301 might be, 
for example, a viewer utility by which a user selects certain streaming data for 
30 presentation. A program or other module that requests or specifies a stream of data will 
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be referred to as a client. Program 302 represents a software module for transforming 
data in some way, such as a software digital filter or compander. Device 303 could be a 
hardware module such as a memory or an MPEG-2 decoder. Device 304 might represent 
an off-line storage device such as a DVD player or a cable TV, with its hardware 
5 interface adapter 3 05 . 

Physical memories in system 100 have memory manager components 320 for 
organizing the data stored in them. A single physical memory module can have multiple 
managers for organizing different data at different times or in different parts of the 
module. A single manager can also serve multiple physical memories. The significant 

10 function of managers 320 in the present context is to allocate and deallocate blocks of 
memory for storing frames or other units of streaming data. For this reason, managers 
320 will frequently be referred to as memory allocators herein. A frame is allocated 
whenever newly arriving data requests it, or it can be pre-allocated. The frame carries the 
data through one or more filters in a path, and is deallocated when all filters in the path 

15 have finished processing that data. Frames can be destroyed, but are usually recycled 
with fixrther new data arriving in the path. 

In Windows-2000, an I/O subsystem 330 supervises both file storage and other 
I/O devices and facilities. Requests for file or I/O services are routed from an application 
program or other source to hardware devices such as 303 and 304 via one or more layers 

20 of device drivers such as 331 and 332. Along the way, filter drivers such a s 333 and 334 
may intercept the data, file handles, I/O request packets, and other information, based 
upon certain characteristics or events. Filter drivers can process data internally as shown 
at 333. They can also pass information back and forth to programs such as 302, which 
can be located within the OS kernel layer or at any other point in the software 

25 architecture of system 100. Components can be dedicated to a single function, or, more 
often, can be programmed to carry out multiple fimctions, either sequentially or 
concurrently. A digital signal processor, for example, can execute many different 
fimctions such as frequency filtering, gain changing, and acoustic effects. 

Block 340 represents the WDM-CSA components that build and manage graphs 
30 for streaming data, and includes a graph data-flow manager 341 and a graph control-flow 
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manager 342. Block 340 also includes a pipe manager that builds several objects used by 
the invention, as explained in detail below. Managers 341-343 need not be implemented 
in single software or hardware modules; their functions can easily be distributed among 
many such modules. 

5 Fig. 4A shows one pipe 400 of a graph such as shown in application Docket 

777.183usl for transferring one or more streams of data through selected ones of the 
components 300 in order to achieve one or more overall transformations of the data. 
Individual hardware or software processing modules, usually called filters, are shown as 
light rectangles surrounding heavier function blocks. In the conventions of the WDM- 

10 CSA protocol, functions communicate data to each other by means of logical pins, 
labeled P. For the present purposes, filters are of two broad types. Non-restructuring 
filters perform a function or transformation upon data. Restructuring filters have three or 
more pins, and are capable of splitting or mixing frames of streaming data. Some can 
both mix multiple input frames and spUt a frame into multiple outputs. The term 

15 "restructuring" will be used to refer to mixing, splitting, and other operations that alter 
the frame structure of their data. The term "splitter/mixer" refers to a module that 
performs one or more restructuring operations. Heavy arrows signify the transfer of data 
among the different functions. 

Filter functions that mix or split data frames are becoming much more common as 
20 streaming data becomes more complex. However, these restructuring functions are 
difficult to integrate efficiently into a graph. One of the concepts of apphcation 
777.183US1 is to increase overall efficiency by dividing a graph into pipes. It is difficult, 
however, to incorporate cascaded or multiple splitters and mixers into a single pipe. 
None of the pipes in that application include more than one restructuring function. 

25 Using this invention, the single pipe 400 includes nine filters, six of which are 

restructuring: five mixers and one spHtter. Filter module M4A mixes input streaming- 
data frames arriving at pins P401 and P402 to produce a combined frame at pin P403, 
while module M4B mixes input frames at pins P404 and P405 into a merged frame at 
P406. A further mixer M4C accepts these two frames at pins P407 and P408 and outputs 

30 another merged frame at P409. Modules M4A, M4B, and M4C might also perform some 
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other processing function or transformation, such as signal compression or filtering, in 
addition to mixing. Module M4D is a simple function that processes firames at pin P410 
and outputs them at P411 . Modules M4E and M4F process input frames at pins P412 and 
P414 into the inputs P416 and P417 of mixer M4G, which combines them at P418. 
5 Mixer module M4H accepts fi*ames fi*om P41 1 and P418 at connected input pins P419 
and P420, merging them into a single composite firame at pin P42L Finally, sphtter 
module M4I produces multiple outputs at pins P423 and P424. This module might also 
perform processing in addition to splitting a single firame into subframes. 

Every streaming data pipe includes a memory allocator for allocating and 
10 deallocating memory frames for the streaming data passing through a pipe. The overall 
allocated frame is referred to as a physical frame, because it occupies physical space in a 
memory. Subframes within the overall frame are called virtual frames, in that they live 
within the overall frame, and do not require separate or additional physical memory 
space. Dashed box 410 indicates one position for an allocator 410 for pipe 400; in this 
1 5 example, the allocator is assigned to pin P421 of module M4H. AppUcation 777. 1 83usl 
discusses how to select the position and specifications of an allocator for a given pipe. 
As discussed below, an allocator for the present purposes can allocate more complex 
frames than those for application 777.183usl. 

Fig. 4B symbohzes a complex data frame 420 for use with the example pipe 400 
20 of Fig. 4A. In this description, the term "frame" can refer to any frame, whether or not it 
forms a part of another frame or has other frames within it. The term "composite frame" 
refers to a frame that has or may have other frames nested within it. A "subframe" is a 
frame that is or may be nested within another frame. The composite frame 420 is called 
F421, because it represents the overall data frame handled by allocator 410 located at pin 
25 P421. The two subframes of frame F421 are labeled F419 and F420 because they occur 
atpinsP419 andP420. F4 19 in turn has further subframes F407 and F408. Input frames 
F401, F402 and F404, F405 are subframes of F407 and F408, respectively. The 
subframes of F420 are input frames F412 and F414. 

Fig. 4C shows another representation 430 of the structure of the overall composite 
30 frame for pipe 400. Composite-frame nesting tree 430 has directed edges representing 
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the direction of data flow. There are three possible cases, depending upon the graph- 
relative positions of the pins that are associated with the frames. One pin can be 
upstream, downstream, or at the same level as the other pin. Input subframes F401 and 
F402 flow into a node containing a frame that can be labeled as both F403, referenced to 

5 output pin P403, or as F407, referenced to pin P407. Input subframes F404 and F405 
similarly flow into node F406/F408, Subframes F407 and F408 similarly mix in a frame 
labeled F409, F410, F411, or F419, because the same frame exists at all four of the 
corresponding pins. Likewise, the same subframes F416 and F417 can also be labeled 
F412, F413 and F414, F415, respectively. These are also subframes of F418, also known 

10 as F420. This upstream flow ends at the merging of F419 and F420 into the Ml 

composite frame F421, F422. Downstream data flow from this frill frame proceeds to its 
subframes F423 and F424. 

Streaming data frames through a pipe using a single overall physical frame such 
as F421 requires that stream manager 340, Fig. 3, provide the frame's memory 

15 management and control. This requires resolving the frame allocator's dependencies, 
which can be done by building the nesting trees such as 430. Conventional operating- 
system streaming-data facilities do not provide sufficient constraints to generate a unique 
nesting in some cases. The relevant information provided by WDM-CSA, for example, is 
hmited to the framing properties of individual module pins, the topology of the graph, 

20 and the boundaries of the data pipes. 

Fig. 5 illustrates a problematic situation. In Fig. 5A, pipe 500 includes two simple 
fimctions MSA and MSB and two restructuring fimctions, a mixer/splitter MSG and a 
mixer M5D. The convention for naming pins and frames follows that of Fig. 4. 

One advantage of the invention is that it reduces the number of pipes in many 
25 streaming-data graphs, by avoiding the placement of cascading mixers and splitters in 
separate pipes. However, a single overall graph can still include multiple composite- 
frame pipes according to the invention. In that case, the pipes are joined to each other in 
the same manner as that described in appHcation docket 777.183US1. 

Fig. SB shows one possible layout S 10 of a composite frame serving pipe SOO. 
30 The mixer portion of module MSG mixes subframe FS02 (same as FSOS) with subframe 
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F507 (F504) into composite frame F506 and frame F508. Mixer M5D then combines 
these two frames into another composite frame F510 at output pin F510 of the pipe. 
Nesting-tree diagram 520, Fig. 5C, shows this solution in the same form as trees 430 
represent the frame layout of pipe 400, Fig. 4. 

5 Fig. 5D shows another configuration 530 for the same pipe 500. Frame layout 

531 has subframes F502, F504, and F508 nested in a composite frame F506 at a point 
where module M5C has finished mixing them, but before module M5D begins merging 
its input data at pins P509 and P51 1 . Nesting tree 540 in Fig. 5E demonstrates that 
module M5D need not actually allocate more memory for its output composite frame 
10 F5 1 0 than the memory required for its inputs F509 and F5 1 1 ; it can output the mixed data 
directly into composite frame F509 (same as frame F506). 

Thus, a first configuration, Figs. 5B and 5C, has frame F510 as the overall 
physical frame allocated by ftmction module M5D for pipe 500. In a second 
configuration, Figs. 5D and 5E, module M5C allocates frame F506 as the single overall 
15 physical frame. Both configurations satisfy the graph topology and the boundaries of 
pipe 500. Therefore, the general case necessitates additional constraints in order to 
specify frame allocators for composite frames. Stream manager 340, which has access to 
the overall graph functions, can supply this information in the form of additional 
composite-frame nesting trees. 

20 The first objective in improving streaming data flow is to assign an allocator to 

each pipe to manage the overall frame for that pipe. Application 777.183US1 discusses 
the operation of memory allocators 320, Fig, 3, for simple frames, and how to position 
them within pipes. The present invention extends such allocators to handle complex 
nested frames. 

25 Fig. 6 recapitulates the overall data-flow process of application 777.183US1, and 

adds a method for improving the flow of composite-frame data. Block 601 of Fig. 6A 
assigns values of certain parameters to each function unit in a streaming-data graph. In 
block 602, the user, appUcation program, or other chent software sets overall goals for 
the graph. 
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Block 603 constructs the desired graph, including defining a preliminary set of 
pipes. Blocks 610 perform additional operations for composite frames. Block 611 
collects parameter data assigned to the filters in block 602 concerning subframes from the 
modules of the graph. One parameter is the filter-suggested pipe topology: the 
5 association between pins and pipes for each filter. A filter expresses this information as 
optional properties for mixers and spHtters, as the set of lists of the filter's pins that 
correspond to a particular pipe. For example, mixer M4G5 Fig. 4, specifies that pins 
P416, P417, and P418 can belong to a single pipe. A filter can specify multiple 
topologies in order of their preference. The WDM-CSA stream manager 340 selects one 

10 pipe topology for each filter from the filter-suggested fist when it builds the pipes for the 
whole graph. A "constant offsef flag can promise that a filter will maintain a subframe 
at the same position within its composite frame at all times when the graph is played. 
This flag is set as a framing property of the filter pin associated with the subframe. If the 
flag is set, a subframe-offset value specifies the constant offset. (These two can be 

15 combined; e.g., a negative offset value can act as a flag that the subframe offset is not 
guaranteed to be constant.) The offset value is a number of bytes relative to the 
composite frame address of the same filter. Graph manager 340 will compute the 
subframe addresses relative to the actual physical frame of its pipe from the pipe-wide 
nesting tree and each filter's subframe offsets. In a very few cases, such as when the 

20 graph cHent requires unusual memory transforms, an additional nesting tree must be 
supplied; this is the situation depicted in Figs. 5D and 5E. 

Block 612 construct a unique nesting tree for each pipe that has a composite 
frame. When an application 112 sets up to play a graph, it acquires the graph at block 
604. 

25 Block 605 finalizes the pipes. Blocks 620 set up composite frames within the 

overall process of block 605. Block 621 selects the sizes of the overall frame and of any 
subframes, if it is composite. 

Block 622 determines whether or not the pipe has any cascaded mixers, such as 
M4A-M4C-M4D-M4H and M4G-M4H in Fig. 4A.. If so, block 623, performed by 
30 pipe manager 343, Fig. 3, constructs two tables 700 specifying the layout of a composite 
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frame. As shown in Fig. 7A, offset table 710 has an entry 71 1 for each pin of each 
module in a pipe and a colunm 712 listing an offset amoimt for each entry. The 
illustrative entries in Fig. 7 follow the example pipe 400 shown in Fig, 4. The subframe 
F401 at pin P401 in Fig. 4A is offset from the beginning of the overall composite frame 
5 F421 by a byte amount symbolized by the quantity Offset #01, and similarly for the other 
subframes. The subframes at some pins are of course the same as those at other pins, so 
that multiple ones of the entries have the same offset. Pipe control table 720 in Fig. 7B 
has an entry 721 for each filter module in the pipe. Column 722 hsts other modules in 
the same pipe that source data frames to that module. For example^ module M4C 
10 receives streaming data from filter modules M4A and M4B. Modules in "Start" section 
723 accept data from outside the pipe. Entries 724 indicate dependent modules. Tables 
710 and 720 need not be rebuilt or modified after the graph is fmahzed. The use of these 
tables for improving the control flow of the streaming graph is discussed below, in 
connection with Fig. 6B. 

15 Blocks 630 position the pipe's memory allocator. If 63 1 finds any mixers within 

the pipe, block 632 assigns the allocator to the output pin of the mixer farthest 
downstream in the data flow. In example pipe 400, this is pin P421 of module M4H. If 
block 633 fmds that the pipe contains no mixers but does include sphtters, block 634 
assigns the allocator to the input pin of the splitter farthest upstream in the pipe. In 

20 example pipe 400, this would be pin P422 of module M4L If there are no mixers or 
splitters, block 635 assigns an allocator as described in application 777.183US1. 

The graph is ready to play — that is, ready to process streaming data through its 
modules — in block 606 of Fig, 6B. 

Control of the graph during play can employ the mechanisms described in 
25 copending application docket 777. 1 84US1 , using I/O request packets (IRP) sent from one 
pin to another to send a data frame and to return completed frames. 

Where a pipe has cascaded mixers as described above, the invention not only 
improves the data flow, but can also improve the control flow to reduce processing 
overhead when the graph is played. An important aspect of streaming graphs is to 
30 minimize the overall number of control transactions required to manage the flow of 
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composite frames for propagating streaming data in a pipe. Control is already efficient 
for cascading splitters in a pipe, because data and control flow in the same direction. For 
cascading mixers within a pipe, however, data flow and control have opposite directions, 
because frames divide into subframes in an upstream direction, toward the sources of the 
5 data. Conventional streaming-graph control would begin control transactions at the root 
of the composite-frame nesting tree, and propagate upstream relative to the data flow. 
This can create a significant amount of overhead. For example, the entire pipe 400 of 
Fig. 4 must propagate an I/O request packet (IRP) or other control transaction through all 
the filters before it can commence pulling data from any of the sources M4A, M4B, M4E, 
10 and M4F into subframes F401, F402, F404, and F405. A conventional control sequence 
for pipe 400 would proceed as follows: 

• M4H allocates physical frame F421 and splits it to allocate frames F419 and F420. 

• M4H issues IRP to upstream filter M4D, passing the offset of F419. 

• M4H issues IRP to upstream filter M4G, passing the offset of F420. 
15 • M4D issues IRP to upstream filter M4C, passing the offset of F419. 

• M4G splits F420 to allocate frames F416 and F417. 

• M4G issues IRP to upstream filter M4E, passing the offset of F416. 

• M4G issues IRP to upstream filter M4F, passing the offset of F417. 

• M4C splits F419 to allocate frames F407 and F408. 

20 • M4C issues IRP to upstream filter M4A, passing the offset of F407. 

• M4C issues IRP to upstream filter M4B, passing the offset of F408. 

Therefore, filters M4C, M4D, M4H, and M4G must process each subframe twice: once to 
allocate a subframe, and again to process the data. When any of a set of upstream filters 
completes its subframe, the issuing filter must check whether its other upstream filters 
25 have finished. For example, if M4A returns completed subframe F407 to M4C, M4C 
must check whether its subframe F408 is complete. 
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With the present system of composite frames, a pipe can support multiple 
subframes allocated and circulating simultaneously. In the common case where 
subframes maintain a fixed relationship to the overall frame, it is possible to improve 
control flow within a pipe by performing just-in-time control transactions. Setting the 
5 previously mentioned constant-offset flag indicates that no synchronization or custom 
management is required, and that the offsets of the subframes from the beginning of the 
composite frame never change. 

Fig. 6B shows how to improve flow control in a pipe having constant-offset 
subframes by avoiding the traversal of the entire pipe and the issuance of a control 

10 transaction at every transaction time, that is, whenever any filter completes a subframe, 
whether or not that completion allows another further processing to occur. When the 
filters in a pipe have their constant-offset flags set, blocks 640-660 are performed to 
execute transactions only at times necessary for the actual flow of data. In addition, the 
frame status information recorded in these blocks provides clear and simple pipe-wide 

1 5 run-time context information that is helpful to those who design and maintain streaming 
systems, which are complex, asynchronous, and distributed. Fig. 6B shows the operation 
of only a single pipe; often a graph contains multiple pipes running at the same time. 

Blocks 640 execute asynchronously for each frame in a pipe, as symbolized at 
block 641 ; a pipe can have multiple frames circulating at the same time. Whenever block 

20 642 detects that the pipe needs a new frame, block 643 causes allocator 410, Fig. 4 to 
create it. Block 644 builds and initializes a new control table for that frame. Fig. 8 
shows frame control tables 800 for several active frames. Table 810, associated by tag 
81 1 with a first circulating frame, has an entry 812 for each filter module M4A-M4I, 
again using pipe 400 as an illustrative example. Column 813 contains a flag for each 

25 entry, indicating whether or not each module has processed that particular frame. Table 
820, associated by tag 821 with another frame in circulation at the same time, has the 
same entries 822, although of course the values of flags 823 will usually differ. A newly 
created table has all flags cleared. 

Blocks 650 are performed by each filter module listed in the "Start" section 723 
30 of pipe control table 720, Fig. 7, as symbolized at block 651 . After blocks 640 have 
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issued a new frame, each start module begins a data-sourcing operation 652 to fill its 
subframe of the new frame from outside the pipe — from another pipe or from a storage or 
other hardware device. 

Blocks 660 are executed repeatedly and asynchronously by each pipe module, as 
5 the filters perform their operations. Whenever block 661 finds that a particular filter has 
completed its operation on a frame or subframe, block 662 places a completion flag 813 
or 823 in the entry 812 or 822 of the table 810 or 820 for that circulating frame. For each 
entry 721 in which it appears as a source in column 722, block 663 causes a module to 
check the frame control table to determine whether all other modules that supply data to 
10 the same downstream module have completed their operations on that frame. If block 
664 finds that all these other modules have completed their operations on that frame (i.e., 
if the completion flags for their entries are set), then block 665 initiates a control 
transaction for the next module downstream in the pipe. That is, a downstream module is 
called only when all of its data has become available. 

15 For example, when module M4B, Fig. 4, has completed acquiring and processing 

subframe F406, it sets its completion flag in the appropriate frame control table. Then it 
looks at the pipe control table and finds that downstream filter M4C also depends upon 
subframe F403 from module M4A. Module M4B reads the same frame control table to 
determine whether module M4A had already reported completion of subframe F403 by 

20 setting the flag for its frame control table entry. If so, M4B initiates a control transaction 
to M4C to begin mixing frames F403 and F406, Otherwise, no control transaction occurs 
at that time. Then, at a later time when M4A completes processing subframe F403 and 
sets the M4A flag, it looks and finds that M4B has already set its flag. Therefore, M4A 
calls module M4C. That is, the last module to finish a subframe required by a 

25 downstream module is the only one that initiates a control transaction for that module. 

As stated earlier, this method is carried out for every pipe in a graph that has 
composite frames. In the most general case, a multiple pipes can pass through the same 
filter; to accommodate this situation, it is possible to base the frame control tables and the 
pipe control tables upon individual module pins, rather than upon the filters themselves. 
30 Any inter-pipe dependencies, such as data-size requirements, time synchronization, 
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resource usage, overall graph goals, and so forth, are addressed by the pipe managers 
among themselves in a conventional manner. 

Conclusion 

5 The present invention improves the data flow and the control of streaming data 

through a graph having splitter/mixer processing modules for restructuring the data as 
well as for transforming it. 

Although only splitters and mixers have been discussed specifically, they can be 
considered as instances of a broader class of restructuring operations in which the 

10 invention also finds utility. For example, a sending filter might transfer multiple 

subframes to a receiving filter through a single output pin. One practical application is a 
silence-compression filter having a single input pin and a single output pin. Such a filter 
receives a fi*ame of audio data at its input pin, parses the data to skip all the silent periods, 
then passes the non-silent segments to a downstream filter via the single output pin. This 

15 subgraph — the silence compressor and its downstream consumer — can be implemented 
using a single pipe with multiple subfi*ames. The input frame of the silence compressor is 
the composite frame, the multiple output subframes of the silence compressor are the 
parts of the input composite frame. Such an implementation reduces the memory used by 
the subgraph, because it employs a single composite frame for the entire subgraph. It can 

20 also reduce the CPU processing load, if much of the data are not overridden by the 
silence compressor. 

Having described preferred implementations and a few extensions and variations 
thereof, we claim as our invention: 
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Claims 



1 . A method for processing frames of streaming data through modules in a digital 
computer, comprising: 

constructing a pipe as a connected group of multiple ones of the modules, at least 
one of the modules being a restructuring module; 

allocating composite frames having predefined subframes; 

transporting the streaming data through different ones of the modules in the group 
in different ones of the subframes; 

restructuring the data among at least some of the subframes in the restructuring 
module. 

2. A computer readable medium bearing instructions and data for causing a digital 
computer to execute the method of claim 1 . 

3. The method of claim 1 where the composite frame is a physical frame in a memory. 

4. The method of claim 3 where the subframes are virtual frames defined in the same 
memory as the physical frame. 

5. The method of claim 1 fiirther comprising assigning an allocator for the composite 
frames to one of the modules in the pipe, 

6. The method of claim 5 where the allocator is assigned to the farthest upstream 
restructuring module in the pipe. 
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7. The method of claim 5 where the allocator is assigned to the farthest downstream 
restructuring module in the pipe, 

8. The method of claim 1 further comprising constructing a frame nesting tree specifying 
how the data is to be restructured in the restructuring module. 

9. The method of claim 1 further comprising: 

repeating the above steps for further pipes including further modules in the 
computer; 

joining the pipes together into a graph. 

10. The method of claim 1 where transporting the streaming data includes issuing a 
control transaction to the restructuring module only when all of the subframes processed 
by that module become available. 

1 1 . A method for processing frames of streaming data through multiple modules disposed 
in a pipe including at least one restructuring module in a digital computer, comprising: 

allocating a composite frame having multiple subframes; 

performing operations upon the subframes in any of the modules sourcing data to 
the restructuring module; 

after completion of the operations for all of the subframes sourcing data to the 
restructuring module, issuing a control transaction to the restructuring module; 

performing operations upon the subframes sourced to the restructuring module in 
response to the control transaction. 

12. A computer readable medium bearing instructions and data for causing a digital 
computer to execute the method of claim 1 1 . 
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13. The method of claim 1 1 where allocating the composite frame includes constructing a 
frame control table having an entry for each module in the pipe and a flag for each of the 
modules indicating whether a particular module has completed an operation upon the 
frame, 

14. The method of claim 13 fiirther comprising setting one of the flags whenever a 
module has completed an operation upon a frame. 

15. The method of claim 1 1 ftirther including constructing an offset table specifying the 
structure of the subframes within the composite frame. 

16. The method of claim 1 1 fiirther comprising constructing a pipe control table 
specifying the structure of the modules in the pipe. 

17. The method of claim 16 where the pipe control table has an entry for at least some of 
the modules in the pipe, and where each entry specifies which other module or modules 
source data to one of the modules in the pipe. 

18. The method of claim 16 where the pipe control table has an entry for each module in 
the pipe that sources data from outside the pipe. 

19. The method of claim 1 1 ftirther comprising repeating the above steps overlapped in 
time such that a plurality of composite frames circulate within the pipe concurrently. 

20. The method of claim 19 fiirther comprising constructing a separate frame control 
table for each of the composite frames. 
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21 . A method for processing frames of streaming data through modules including 
multiple restructuring modules in a digital computer, comprising: 

constructing a pipe as a connected group of the modules including the 
restructuring modules; 

assigning a single allocator to one of the modules; 

allocating composite frames having predefined subframes associated with 
respective ones of the restructuring modules; 

transporting the streaming data through different ones of the modules in the group 
in different ones of the subframes; 

restructuring the data among the subframes in the restructuring modules. 

22. A computer readable mediimi bearing instructions and data for causing a digital 
computer to execute the method of claim 21 . 

23. The method of claim 21 fiirther comprising constructing a frame nesting tree 
specifying how the data is restructured by the restructuring module. 

24. The method of claim 21 fiirther comprising collecting a constant-offset flag for each 
module. 

25. The method of claim 24 fiirther comprising collecting an offset value for any module 
whose constant-flag is set. 

26. The method of claim 24 fiirther comprising constructing an offset table specifying 
relationships of the subframes to the composite frame. 
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27. The method of claim 24 further comprising specifying memory sizes for each of the 
subframes within the composite frame. 

28. The method of claim 21 where a plurality of the restructuring modules are of a single 
type in a cascade in the pipe. 

29. The method of claim 28 where the allocator is assigned to a particular module in 
response to the type of cascaded modules. 

30. The method of claim 28 where the plurality of cascaded modules are mixers, and 
where the allocator is assigned to a downstream one of the cascaded mixers. 

31. The method of claim 28 where the plurality of cascaded modules are sphtters, and 
where the allocator is assigned to an upstream one of the cascaded spHtters. 

32. The method of claim 21 where transporting the streaming data includes issuing 
control transactions to the restructuring modules only when all of the subframes 
processed by respective ones of the modules become available. 

33. A method for processing frames of streaming data through modules including 
multiple restructuring modules connected in a pipe in a digital computer, comprising: 

allocating a composite frame having multiple subframes for different ones of the 
restructuring modules; 

performing data-sourcing operations upon certain of the subframes in source ones 
of the modules; 
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when each one of the data-sourcing operations has completed, determining 
whether one of the restructuring modules has all of the subJframes required for it to 
perform an operation; 

if so, issuing a control transaction to the one restructuring module; 

performing an operation in the one restructuring modules after receiving a control 
transaction, 

34. A computer readable medium bearing instructions and data for causing a digital 
computer to execute the method of claim 33. 

35. The method of claim 33 where the source ones of the modules are those receiving 
data from outside the pipe. 

36. The method of claim 33 where a plurality of the restructuring modules are mixers. 

37. The method of claim 36 where a plurality of the mixers are cascaded in the pipe. 

38. The method of claim 33 further comprising: 

when each one of the data-sourcing operations has completed, determining 
whether others of the restructuring modules have all of the subframes required for them 
to perform operations; 

if so, issuing control transactions to the other restructuring modules; 

performing operations in the other restructuring modules after receiving the 
control transactions, 

39. The method of claim 33 fiirther comprising: 
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storing a separate completion flag for each of the modules in the pipe; 

setting one of the completion flags when a corresponding one of the modules has 
completed an operation upon the frame. 

40. The method of claim 39 where the determining step comprises: 

reading a table listing all of the other modules that source data to the one 
restructuring module; 

determining whether a completion flag for all of the other modules has been set, 

41. The method of claim 33 further comprising repeating the above steps overlapped in 
time such that a plurality of composite frames circulate within the pipe concurrently, 

42. The method of claim 41 further comprising constructing a separate frame control 
table for each of the composite frames, 

43. A data structure for processing streaming data through multiple restructuring 
processing modules in a graph, comprising: 

a single composite frame physically allocated in a memory; 

a plurality of virtual subframes allocated within the composite frame. 

44. The structure of claim 43 where different ones of the subframes contain different 
types of data. 

45. The structure of claim 44 where one type of data is video data. 

46. The structure of claim 44 where another type of data is audio data. 
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47. The structure of claim 43 further comprising an offset table for locating the subJframes 
with respect to the composite frame. 

48. The structure of claim 47 where the offset table includes an entry for each of the 
subframes and a column for specifying the locations of the sub frames within the 
composite frame. 

49. The structure of claim 43 fiirther comprising a pipe control table representing the 
structure of a pipe of modules for processing streaming data in the data structure. 

50. The structure of claim 49 where the pipe control table includes an entry for each of 
the modules in the pipe. 

51. The structure of claim 50 where the pipe control table includes a source column for 
listing other modules that source data to the modules in each of the entries. 

52. The structure of claim 50 where the pipe control table includes a start section listing 
those modules that source data from outside the pipe. 

53. The structure of claim 43 ftirther comprising a frame control table for tracking the 
status of the frame with respect to the processing modules. 

54. The structure of claim 53 where the frame control table includes an entry for each of 
the modules in the pipe. 
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55. The structure of claim 54 where the frame control table includes a column for holding 
flags indicating that each of the modules has completed processing the frame. 

56. The structure of claim 53 further comprising a separate frame control table for each 
frame circulating in the pipe at a particular time. 

57. A computer system for processing streaming data, comprising: 

a plurality of modules for processing the streaming data, at least some of which 
are restructuring; 

a plurality of memory managers for allocating composite frames containing 
subframes for containing streaming data; 

a flow maaager for constructing a graph having at least one pipe including a 
plurality of the restructuring modules, and for assigning one of the memory managers to 
the pipe. 

58. The system of claim 57 fiirther comprising a memory for storing the composite 
frames. 

59. The system of claim 58 where the subframes are allocated within the composite 
frames in the memory. 

60. The system of claim 58 fiirther including a processor. 

61. The system of claim 60 where the processor implements one or more of the 
restructuring modules. 
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62. The system of claim 60 further including an input/output system. 

63. The system of claim 62 where the input/output system implements one or more of the 
modules, 

64. The system of claim 57 further comprising a pipe manager for constructing 
representations of the structures of the composite frames. 

65. The system of claim 64 where one of the representations specifies locations of the 
subframes with respect to the composite frame. 

66. The system of claim 64 where one of the representations specifies relationships of the 
modules within the pipe. 

67. The system of claim 57 further including a control manager for transporting the 
streaming data through the modules in various ones of the subframes. 

68. The system of claim 67 where the control manager issues a control transaction to one 
of the restructuring module for initiating processing of a subframe therein only when all 
of the subframes processed by that module become available. 

69. A computer system for processing streaming data, comprising: 

a plurality of modules disposed in a pipe for processing the streaming data, at 
least some of the modules being restructuring; 

a plurality of memory managers for allocating composite frames containing 
subframes for containing streaming data, different ones of the subframes being associated 
with different ones of the modules; 
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a control manager for issuing control transactions for initiating processing 
operations in the modules. 

70. The system of claim 69 where the control manager issues one of the control 
transactions to a particular module only when certain of the subframes associated with 
that module become available. 

71. The system of claim 70 where the certain subfirames include all of the sub frames that 
source data to the particular module. 

72. The system of claim 69 where certain of the modules receive data from outside the 
pipe. 

73. The system of claim 72 where the control manager issues a control transaction to the 
certain modules when one of the composite frames has been allocated. 

74. The system of claim 69 where at least some of the restructuring modules are mixers 
for combining multiple ones of the subframes. 

75. The system of claim 74 where a plurality of the mixers are cascaded in the pipe. 

76. The system of claim 75 where the control manager issues one of the control 
transactions to any of the mixers only when all of the subframes combined by that mixer 
become available. 

77. The system of claim 69 further comprising a memory for storing the composite 
frames. 
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78. The system of claim 77 where the processor implementing one or more of the 
restructuring modules. 

79. The system of claim 78 further including an input/output system implementing one or 
more of the modules. 

80. A computer readable medium bearing instructions and data for causing a digital 
computer to execute a method for processing frames of streaming data through modules 
in a digital computer, the method comprising: 

constructing a pipe as a connected group of multiple ones of the modules, at least 
one of the modules being restructuring; 

allocating composite frames having predefined subframes; 

transporting the streaming data through different ones of the modules in the group 
in different ones of the subframes; 

restructuring the data among at least some of the subframes in the restructuring 
module. 

SLA computer readable medium bearing instructions and data for causing a digital 
computer to execute a method for processing frames of streaming data through multiple 
modules disposed in a pipe including at least one restructuring module in a digital 
computer, the method comprising: 

allocating a composite frame having multiple subframes; 

performing operations upon the subframes in any of the modules sourcing data to 
the restructuring module; 

after completion of the operations for all of the subframes sourcing data to the 
restructuring module, issuing a control transaction to the restructuring module; 
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performing operations upon the subframes sourced to the restructuring module in 
response to the control transaction. 
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Abstract of the Disclosure 

Streaming data is processed through one or more pipes of connected modules 
including mixers and/or splitters. The data is carried in composite physically allocated 
frames having virtual subframes associated with different ones of the splitters, mixers, 

5 and other transform modules. Nesting trees and pipe control tables represent the structure 
of the pipes. A frame allocator is assigned to a particular module in a pipe. Rather than 
issuing a control transaction to all modules when any one of them completes an operation 
upon its source data, a control manager requests a module to begin its operation only 
when all of its input subframes have become available. Frame control tables record when 

10 any module has completed an operation, and a pipe control table lists which modules 
provide data to which other modules. 
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SCHWEGMAN, LUNDBERG, WOESSNER & KLUTH, P.A. 



United States Patent Application 

COMBINED DECLARATION AND POWER OF ATTORNEY 

As a below named inventor I hereby declare that: my residence, post office address and citizenship are as stated below next to my 
name; that 

I verily believe I am the original, first and joint inventor of the subject matter which is claimed and for which a patent is sought on 
the invention entitled: EFFICIENT SPLITTING AND MIXING OF STREAMING-DATA FRAMES FOR PROCESSING 
THROUGH MULTIPLE PROCESSING MODULES . 

The specification of which is attached hereto. 

I hereby state that I have reviewed and imderstand the contents of the above-identified specification, including the claims, as 
amended by any amendment referred to above. 

I acknowledge the duty to disclose information which is material to the patentability of this application in accordance with Title 37, 
Code of Federal Regulations, ^ 1.56 (see page 4 attached hereto). I also acknowledge my duty to disclose all information known to be 
material to patentability which became available between a filing date of a prior application and the national or PCT international filing date 
in the event this is a Continuation-In-Part application in accordance with Title 37, Code of Federal Regulations § 1.63(e). 

I hereby claim foreign priority benefits under Title 35, United States Code, ^11 9/3 65 of any foreign application(s) for patent or 
inv Wor's certificate listed below and have also identified below any foreign application for patent or inventor's certificate having a filing 
date Sefore that of the application on the basis of which priority is claimed: 
No bkch claim for priority is being made at this time, 

ill I hereby claim the benefit under 35 U.S.C. § 1 19(e) of any United States provisional appiication(s) listed below. 

No gach claim for priority is being made at this time, 

Q I hereby claim the benefit under Title 35, United States Code, § 120/365 of any United States and PCT international apphcation(s) 
listQ^below and, insofar as the subject matter of each of the claims of this application is not disclosed in the prior United States application 
in t|g manner provided by the first paragraph of Title 35, United States Code, § 1 12, 1 acknowledge the duty to disclose material information 
as cfefined in Title 37, Code of Federal Regulations, § L 5 6(a) which occurred between the filing date of the prior application and the national 
or Pfij intemational filing date of this application. 

No Mch claim for priority is being made at this time. 



I hereby appoint the following attomey(s) and/or patent agent(s) to prosecute this application and to transact all business in the 
Patent and Trademark Office connected herewith: 



Adams, Matthew W. 


Reg. No. 


43,459 


Forrest, Bradley A. 


Reg. 


No- 


30,837 


Mates, Robert E. 


Reg. No. 


35,271 


Anglin, J. Michael 


Reg. No. 


24,916 


Harris, Robert J. 


Reg. 


No, 


37,346 


McCrackin, Ann M. 


Reg. No. 


42,858 


Arora, Suneel 


Reg. No. 


42,267 


Holloway, Sheryl S. 


Reg. 


No. 


37,850 


Nielsen, Walter W. 


Reg. No. 


25,539 


Bianchi, Timothy E. 


Reg. No. 


39,610 


Huebsch, Joseph C. 


Reg. 


No. 


42,673 


Oh, Allen J. 


Reg. No. 


42,047 


Bilhon, Richard E, 


Reg. No. 


32,836 


Jurkovich, Patti J. 


Reg. 


No. 


42,444 


Padys, Danny J. 


Reg. No- 


35,635 


Black, David W. 


Reg. No. 


42,331 


Kalis, Janal M. 


Reg. 


No. 


37,650 


Polglaze, Daniel J. 


Reg. No. 


39,801 


Brennan, Thomas F. 


Reg No. 


35,075 


Klima-Silberg, Catherine I. 


Reg. 


No. 


40,052 


Sako, Katie E. 


Reg. No. 


32,628 


Brooks, Edward J., Ill Reg. No. 


40,925 




Kluth, Daniel J. 


Reg. 


No. 


32,146 


Schwegman, Micheal L. 


Reg. No. 


25,816 


Clark, Barbara J. 


Reg. No. 


38,107 


Lacy, Rodney L. 


Reg. 


No. 


41,136 


Sieffert, Kent J. 


Reg. No. 


41,312 


Crouse, Daniel D. 


Reg. No. 


32,022 


Leffert, Thomas W. 


Reg. 


No. 


40,697 


Slifer, Russell D. 


Reg. No. 


39,838 


Drake, Eduardo E. 


Reg. No. 


40,594 


Lemaire, Charles A 


Reg. 


No. 


36,198 


Steffey, Charles E. 


Reg. No. 


25,179 


EUseeva, Maria M. 


Reg. No. 


43,328 


Litman, Mark A. 


Reg 


No. 


26,390 


Terry, Kathleen R. 


Reg. No. 


31,884 


Embretson, Janel E 


Reg No. 


39,665 


Lundberg, Steven W. Reg. No. 


30,568 




Viksnins, Ann S. 


Reg. No. 


37,748 


Fogg, David N. 


Reg. No- 


35,138 


Mack, Lisa K. 


Reg. 


No. 


42,825 


Woessner, Warren D. Reg. No. 30,440 




Fordenbacher, Paul J. Reg. No. 


42,546 




Maki, Peter C. 


Reg No. 


42,832 









I hereby authorize them to act and rely on instructions from and communicate directly with the 
person/assignee/attomey/fmn/organization/who/which first sends/sent this case to them and by whom/which I hereby declare that I have 
consented after full disclosure to be represented unless/until I instmct Schwegman, Lundberg, Woessner & Kluth, P.A. to the contrary. 



OurRef.777.241USl 
Serial No. not assigned 
Filing Date: not assigned 
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I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information 
and belief are beheved to be true; and further that these statements were made with the knowledge that willful false statements and the 
like so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the vahdity of the application or any patent issued thereon. 

Please direct all correspondence in this case to Schwegman, Lundberg, Woessner & Kluth, P.A. at the address indicated below: 

P.O. Box 2938, Minneapolis, MN 55402 
Telephone No. (612)373-6900 



Our Ref. 777.24 lUSl 
Serial No. not assigned 
Filing Date: not assigned 
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I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information 
and belief are believed to be true; and ftirther that these statements were made with the knowledge that willful false statements and the 
like so made are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States Code and that such 
willful false statements may jeopardize the vahdity of the application or any patent issued thereon. 



Full Name of joint inventor number 1 : Rafael S. Llsitsa 
Citizenship: United States of America 

Post Office Address: 443 1 145 Avenue SE 

Bellevue, WA 98006 



Signature: 



Rafael S. Lisitsa 



Residence: Bellevue, WA 




Date: 



Full Name of joint inventor number 2 : George H. J. Shaw 
Citizenship: United States of America 

Post Office Address: 21213 NE 186th Street 

Woodmville, WA 98072 



Sig]^|t:ure: 



George H. J, Shaw 



Fulf §Jame of joiat inventor number 3 : Dale A. Sather 
Citranship : United States of America 

Posl^ffice Address: 6813 26fh Avenue NE 

% Seattle, WA 98115 



Sigiia^ture: 



Dale A. Sather 



Residence: Woodinville, WA 





Residence: Seattle, WA 



Date: 



Fuli&ame of joint inventor number 4 : Bryan A. Woodruff 
CitiSnship: United States of America 

PosH)ffice Address: 1020 SW 10th Street 

North Bend, WA 98045 



Signature: 



Residence: North Bend, WA 




OurRef. 777.241US1 
Serial No. not assigned 
Filing Date: not assigned 
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§ 1.56 Duty to disclose information material to patentability. 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, and the most effective patent 
examination occurs when, at the time an appHcation is being examined, the Office is aware of and evaluates the teachings of all information 
material to patentability. Each individual associated with the filing and prosecution of a patent application has a duty of candor and good 
faith in dealing with the Office, which includes a duty to disclose to the Office all information known to that individual to be material to 
patentability as defmed in this section. The duty to disclose information exists with respect to each pending claim until the claim is canceled 
or withdrawn from consideration, or the application becomes abandoned. Information material to the patentability of a claim that is canceled 
or withdrawn from consideration need not be submitted if the information is not material to the patentability of any claim remaining under 
consideration in the application. There is no duty to submit information which is not material to the patentability of any existing claim. The 
duty to disclose all information known to be material to patentability is deemed to be satisfied if all uiformation known to be material to 
patentability of any claim issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§ l,97(b)-(d) 
and 1 .98. However, no patent will be granted on an application in connection with which fraud on the Office was practiced or attempted or 
the duty of disclosure was violated through bad faith or intentional misconduct. The Office encourages apphcants to carefully examme: 

(1) prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) the closest information over which individuals associated with the filing or prosecution of a patent appHcation believe any 
pending claim patentably defmes, to make sure that any material information contained thereia is disclosed to the Office. 

^9 Under this section, information is material to patentability when it is not cumulative to information akeady of record or being 
ma|d of record in the application, and 

O ( 1 ) It establishes, by itself or in combination with other information, a pruna facie case of unpatentability of a claim; or 

In (2) It refutes, or is inconsistent with, a position the apphcant takes in; 

7 (i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A jpma facie case of unpatentability is established when the information compels a conclusion that a claim is unpatentable under the 
prefi^nderance of evidence, burden-of-proof standard, giving each term in the claim its broadest reasonable construction consistent with the 
spec&cation, and before any consideration is given to evidence which may be submitted in an attempt to establish a contrary conclusion of 
pat^abihty, 

(c) Individuals associated with the filing or prosecution of a patent application within the meaning of this section are: 

( 1 ) Each inventor named in the application: 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the appHcation and who is associated 
with the inventor, with the assignee or with anyone to whom there is an obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by disclosing information to the attomey, 
agent, or inventor. 



